Repository navigation
feat(openai): support output logprobs for chat completions - #1701
Merged
Merged
Conversation
Add logprobs and top_logprobs to streaming and non-streaming responses. Use Triton to process requested logits rows directly and skip top-k selection when only the sampled token score is requested. Pack candidates by request and preserve alignment through MTP filtering.
Capture candidates only for rows that request top-k, and hand post-handle a list aligned with the logits rows so accepted tokens can be zipped directly. Co-authored-by: Cursor <cursoragent@cursor.com>
hiworldwzj
force-pushed
the
openai-chat-output-logprobs
branch
from
November 13, 2026 05:07
1009cf4 to
fa8fe0d
Compare
Keep sampled logprob and rank in their own buffer, and give candidate rows a dedicated segment with spare output slots. Co-authored-by: Cursor <cursoragent@cursor.com>
Drop the top-k flag and keep the prefill-sampled token ahead of decode output buffered while waiting for it. Co-authored-by: Cursor <cursoragent@cursor.com>
Pass tensors and Triton constexpr values by name, and annotate the new logprob helpers. Co-authored-by: Cursor <cursoragent@cursor.com>
Keep each output logprob on the reasoning or content text the caller receives, and drop think tags and the stop token. Co-authored-by: Cursor <cursoragent@cursor.com>
Keep the real sampled score unless twenty logits are strictly higher, match reasoning and tool delimiters as token sequences, and reject an invalid sequence before generation starts. Co-authored-by: Cursor <cursoragent@cursor.com>
hiworldwzj
force-pushed
the
openai-chat-output-logprobs
branch
from
November 13, 2026 11:46
297bc52 to
03404f2
Compare
Recover ByteLevel candidate bytes from the token piece, allow a tool parser with an empty end marker, and match gpt-oss channel names by prefix. Co-authored-by: Cursor <cursoragent@cursor.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
/v1/chat/completions支持logprobs和top_logprobs，流式、非流式均返回逐 token 分数及 0–20 个候选。候选按请求的 K 打包并异步拷到 CPU，沿用 MTP 接受掩码和 PD 实际采样节点的首 token 传递路径。采样排名与候选数量独立，K=0 也计算排名。内部保留真实 logprob，仅在 Chat 响应中对排名超过 20 的 token 返回
-9999。ByteFallback 保留每个 token 的原始字节；普通候选用实际 prompt 后缀及已生成上下文解码，扣除未完成字节所贡献的历史文本。EOS 等特殊候选保留身份，不受正文skip_special_tokens影响。API 只解析一次，正文、reasoning、工具调用和 logprobs 使用同一次解析产生的源区间。概率缓冲区不再创建解析器，只保存 token 与位置并按输出分支取出或丢弃分数。非流式工具判定、reasoning 裁剪、流式工具后的空白抑制以及最终 flush 均同步作用于概率，开启 logprobs 不会额外把普通 JSON 当成工具调用。
本轮最终 CPU 回归 655 项通过、7 项排除。新增四类问题的 9 个定向用例在原始
d651699上全部失败，修复后通过；覆盖 logprobs 开关对照、默认 llama3、真实 ByteFallback detokenizer、特殊候选、reasoning 与工具组合及 flush。排除的两个 PD 日志捕获测试族此前已在未修改源码上确认 6 项失败、1 项通过，日志已输出但caplog未捕获。固定版本 Black、Flake8 和git diff --check通过。本轮未改 GPU 采样代码，未重跑 CUDA kernel 或完整模型 MTP/PD 集成；此前
5e4ef99在 H200 上的 61 项采样测试通过。Chat API 回归使用模拟生成序列。相关讨论 #1124。